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[57] ABSTRACT 

A system comprising a neural network, or computer, imple- 
menting a feature detection and a statistical procedure, 
together with fuzzy logic for solving the problem of recog- 
nition of faces or other objects) at multiple resolutions is 
described. A plurality of previously described systems for 
recognizing faces (or other objects) which use local auto- 
correlation coefficients and linear discriminant analysis are 
trained on a data set to recognize facial images each at a 
particular resolution. In a second training stage, each of the 
previously described systems is tested on a second training 
set in which the images presented to the previously 
described recognition systems have a matching resolution to 
those of the first training set, the statistical performance of 
this second training stage being used to train a fuzzy 
combination technique, that of fuzzy integrals. Finally, in a 
test stage, the results from the classifiers at the multiple 
resolutions are combined using fuzzy combination to pro- 
duce an aggregated system whose performance is higher 
than that of any of the individual systems and shows very 
good performance relative to all known face recognitor 
systems which operate on similar types of training and 
testing data, this aggregated system, however, not being 
limited to the recognition of faces and being able to be 
applied to the recognition of other objects. 
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METHOD OF RECOGNIZING IMAGES OF 
FACES OR GENERAL IMAGES USING 
FUZZY COMBINATION OF MULTIPLE 
RESOLUTIONS 

5 

BACKGROUND OF THE INVENTION 

1. Field of the invention 

The present invention relates to a method of recognizing 
images of faces or other images. 10 

2. Description of the Related Art 

The requirements for a general face or image recognition 
system will first be explained. Given a series or several 
series of digitized images of multiple faces (or other objects) 
to be recognized, the system first analyzes or acquires 15 
knowledge of these images in some way. This is called the 
training stage. The training stage allows the system to store 
a characterization of the images which is sufficient to 
distinguish these images from other images of other faces 
(or other objects) and from each other if the characterization 20 
is a good one. The initial series of images are grouped into 
sets corresponding to faces (or other objects) belonging to 
distinct persons (or instances of objects). In some cases, the 
characterization of the image is insufficient to perform the 
identification of the images and reclassification may result. 25 
The training stage may itself be divided into several distinct 
phases. There follows a testing stage. Supplied with a single 
new image or a series of new images of faces (or other 
objects) which may or may not have been previously made 
known to the system, the system is required to state for each 30 
of the new images whether these new images are of faces (or 
other objects) shown to it in the training stage and if so, to 
which one of these faces (or other objects) does the image 
bear the greatest resemblance. If there is insufficient simi- 
larity between the test image or test images and the images 35 
characterized by the system in the training stage, the system 
is required to reject those faces (or other objects) having a 
low degree of similarity to those known to it, such images 
of faces (or other objects) being referred to as unknown. The 
present invention is designed to solve this general face 40 
recognition problem or a problem similar to it in which 
images of objects other than faces are to be recognized. 

For the purposes of discussion, it is convenient to break 
down the list of prior art into five separate sub-lists corre- 
sponding to the overall methodology used. The five topic 45 
areas are: 

1) . the detection of physical features within the face; 

2) . the direct application of neural network technology; 

3) . differencing between sequential video frames; 

4) . forms of traditional image processing; 

5) . applications of fuzzy algorithms to information fusion. 
The method employed in the invention is most closely 

related to topic areas 4 and 5 of this list. Prior art relating to 
the field of face and object recognition will now be dis- 55 
cussed. 

Algorithms and references which are not specifically 
image processing or fuzzy logic related but may be used for 
face recognition or the recognition of objects are now listed 
with some brief explanation. This corresponds to topic areas 60 
1 to 3 in the above list. 

Chow and Xiaobo in "Towards a system for automatic 
facial feature detection" (Pattern Recognition, volume 26 
number 12, pp 1739-1755, 1993) have used the Hough 
transform and deformable template matching to locate facial 65 
features within images. Their technique can be used for 
locating faces within digitized images. Huang, Ming, Ja-ling 
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in "Automatic feature point extraction on a human face in 
model based image coding" (Optical Engineering, volume 
37, number 7, pp 1571-80, 1993) discuss a model-based 
coding scheme which can be used for the extraction of facial 
features such as eyebrows, eyes and facial outlines and could 
then be used for facial recognition. Gordon and Vincent in 
"Application of morphology to feature extraction for face 
recognition" (Proceedings of SPIE, volume 1658, pp 
151-164, 1992) use morphological operators for feature 
extraction of human faces. They describe how to recognize 
connected part boundaries for convex structures as well as 
how to recognize connected ridge lines. Such procedures can 
be used for recognizing facial regions. Stringa in "Eyes 
detection for face recognitor" (Applied Artificial 
Intelligence, volume 7, number 4, pp 365-382, 1993) dis- 
cusses a scheme for eye detection as well as its use in a 
system for recognition of facial images using correlation 
techniques. 

For these techniques, those that can be used for recogni- 
tion lack large scale tests with the exception of the system 
by Stringa. The work of Stringa is used for eye detection and 
the recognition system was based around the technique of 
using differences between successive video frames to deter- 
mine a region of interest followed by histogramming of 
intensity values over a transformed image. This technique 
has the disadvantages of requiring multiple sequential 
images for localizing the region of interest and the technique 
is not directly used for face recognition. 

U.S. Pat. No. 4,641,349 (Flom Leonard and Safir Aran) 
describes a method for identifying eyes, and, by implication, 
persons, animals or objects attached to these eyes. The 
pupils of the eyes are contracted to a predetermined size to 
permit a standard record of the iris to be made. Identification 
then follows from a comparison of the iris images. U.S. Pat, 
No. 5,291,560 (John G. Daugman) describes a method of 
identifying persons using the texture of the iris encoded into 
a compact "iris code'*. Pattern recognition is achieved by 
combining special signal processing methods with statistical 
decision theory. U.S. Pat. No. 5,012,522 (Laurence C. 
Lambert) describes an autonomous face recognition 
machine which is capable of locating human faces in video 
scenes with random content within two minutes, and capable 
of recognizing the faces that it locates. The basic method of 
operation is to identify gestalt faces and to compare the 
gestalt faces' detected facial characteristics with a stored set 
of facial characteristics of known human faces. 

Of these three patents the two concerning recognition 
using information derived from images of irises have the 
obvious limitation that they require special and detailed 
images of the irises which cannot be derived from images 
such as would be produced by typical security cameras 
under normal conditions. The system by Lambert requires a 
sequence of video images and performs the recognition by 
using Gestalt faces. It therefore requires a number of video 
frames to locate the focus of attention and to identify the 
unknown face. 

Rybak, Golovan and Gusakova in "Behavioral model of 
visual perception and recognition" (Proceedings of SPIE, 
volume 1913, pp 548-560, 1993) have used a system 
comprising a low-level neural module for attention focusing 
in conjunction with a high-level recognition module for 
recognizing complex objects such as faces. Solheim, Payne 
and Casta in in "The potential in using backpropagation 
neural networks for facial verification systems" (Simulation, 
volume 58, number 5, pp 306-310, 1992) have used neural 
networks trained on pixel vectors formed from compressed 
facial images for face recognition. Ying, Nakano and Miyao 
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in "The extraction and recognition of facial images in the 
complex background" (Journal of the Faculty of 
Engineering, Shinshu University, volume 74, pp 77-87, 
1994) used a facial texture model whose coefficients were 
derived by perceptron-type learning. Soulie, Viennet and 
Lamy in "Multi-modular neural network architectures: 
applications in optical character and human face recogni- 
tion" (International Journal of Pattern Recognition and Arti- 
ficial Intelligence, volume 7, number 4, pp 721-755, 1993) 
describe the integration of several neural networks into a 
pattern recognition system which is globally trained. The use 
of neural networks for face recognition has been reviewed 
by Valentin, Abdi, O Toole and Cottrell in "Connectionist 
models of face processing: a survey" (Pattern Recognition 
volume 27, number 9, pp 1209-1230, 1994). 

The difficulty in using neural techniques for recognition 
lies in the large number of test images required and the long 
training times used in order to teach the network the image 
information. However, they do have the advantage that they 
are able to adapt to changing information and are able to 
learn new tasks. 

U.S. Pat. No. 5,309,228 (Hiroaki Nakamura) describes a 
method of extracting feature image data and a method of 
extracting a person's face data. An original color image is 
divided into a plurality of elements, resolving these elements 
into red, green and blue components, wherein each of these 
said colors is photometrically measured, obtaining color 
regions of the same or similar hue, inscribing circles or 
ellipses inscribed with respect to the boundary portions of 
the obtained color regions and dividing the color regions 
obtained using circular or elliptic regions starting from the 
largest circle or ellipse in order to thereby obtain a plurality 
of regions and selecting at least one of the divided regions 
from which to extract the data of the selected region as 
human face image data. 

The system by Nakamura relies on the calculation of 
elliptical or circular regions from histogramming data. It is 
likely that this method would be confused in complex 
backgrounds. In any case, once the segmentation is obtained 
variations in the segmentation between different images of 
the same person or object will result in different segmented 
regions, thus affecting the performance of the recognition 
system. Color images of the faces are also required. 

U.S. Pat. No. 5,063,603 (Peter J. Burt) describes a method 
for recognizing objects and an image processing system 
therefor. By examining a time series of images of succes- 
sively higher resolutions in which each frame of the image 
may or may not contain an image of a particular person, the 
earlier low-resolution frames may be used in determining 
areas of interest for characterization at higher resolutions. 
By adding additional data at higher and higher resolutions, 
the system is eventually able to recognize or reject images 
based on thresholding of the probabilities that the image 
does or does not contain a previously identified person. 

The system by Burt requires a sequence of video frames 
in order to locate the person to be identified. In addition, the 
patent is not particularly concerned with the actual mecha- 
nism by which the recognition is performed. 

Algorithms which also apply image processing tech- 
niques to face recognition and have been used for this task 
will now be described with the exception of the algorithm 
closest to that of the invention. This corresponds to topic 
area 4 in the above list. 

Turk and Pentland in "Face processing models for rec- 
ognition*' (Proceedings of SPIE, volume 1192, number 1, pp 
22-32, 1990) discuss the use of eigenfaces for face recog- 
nition. This technique is currently one of the most com- 
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monly used techniques in the field of face recognition. A 
refinement of the eigeuface technique is given by Akamatsu, 
Sasaki, Fukamachi and Suenaga in "A robust face identifi- 
caton scheme — KL expansion of an invariant feature space" 

5 (Proceedings of SPIE, volume 1607, pp 71-34, 1992). 
Bichsel and Pentland have used a scale space matching 
technique for face recognition and tracking in three dimen- 
sions as discussed in " Human face recognition and the face 
image set's topology" (CVGIP Image Understanding, vol- 
umc 59, number 2, pp 254-261 1994). 

The cigenface technique works very well for recognizing 
faces provided that the facial images can be correctly 
aligned and scaled in advance. This problem is well known. 
The scale space matching technique is designed for use with 
sequences of images in order to employ coarse-to-fine 

15 searches in parameter space. The face recognition performed 
is to detect whether or not the presented image is a face or 
some other object. No tests of the discriminatory power of 
the algorithm were presented. 
Hong discusses the use of singular value decompositions 

20 for object recognition in "Algebraic feature extraction of 
image for recognition'* (Pattern Recognition, volume 24, 
number 3, pp 211-219, 1991). His classification procedure 
acts on data whose dimension has been reduced by means of 
the Foley-Sammon transform which is very similar, if not 

25 identical to the technique referred to as linear dfceriminant 
analysis. However, the publication deals only with analysis 
at a single resolution. The works of Cheng, Liu, Yang, 
Yong-Ming, Yan-Chun "Human face recognition based on 
the statistical model of small sample size" (Proceedings of 

30 SPIE, volume 1607, pp 85-95, 1992), Cheng, Liu and Yang 
"A novel feature extraction method for image recognition 
based on similar discriminant function" (Pattern 
Recognition, volume 26, number 1, pp 115-125, 1993) and 
Liu, Jallut, Ying-Jiang, Yoog-Qing, Ying-yu "Novel 

35 approach to human face recognition" (Proceedings of SPIE, 
volume 1771, pp 605-611, 1993) are similar to that pub- 
lished by Hong. 

These techniques are only being used at a single resolu- 
tion. Moreover, their performance with large sample sizes 

40 has not been shown to be effective. 

Takahashi, Sakaguchi, Minami, Nakamura "Description 
and matching of density variation for personal identification 
through facial images" (Proceedings of SPIE, volume 1360, 
number 3, pp 1694-1704, 1990) discuss the use of isoden- 

45 sity lines for human face recognition. Spacek, Kubat and 
Flotzinger "Face recognition through learned boundary 
characteristics" (Applied Artificial Intelligence, volume 8, 
number 1, pp 149-164, 1994) discuss a face recognition 
system which learns to improve its recognition performance. 

50 Samaria and Young in "HMM-based architecture for face 
identification" (Image and Vision Computing, volume 12, 
number 8, pp 537-543, 1994) describe the use of hidden 
Markov models for modeling the statistics of facial images. 
They also show how a hidden Markov model can also be 

55 used for segmenting images and for detecting features that 
can then be used for identification. Much of the previous 
work on face recognition is described by Samal and Iyengar 
in "Automatic recognition and analysis of human faces and 
facial expressions: a survey" (Pattern Recognition, volume 

60 25, number 1, pp 65-77, 1992). 

The use of isodensity lines for recognition requires that 
the image and the test image be similarly scaled and aligned 
and thus suffers from the same problems as the eigenface 
technique. Hidden Markov models are known to be com- 

65 putationally intensive and to take a long time to process. 
Their probabilistic nature means that the eventual segmen- 
tation or recognition result is not deterministic. 
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The U.S. Pat. Nos. 5,063,603, 4,641,349, 5,291,560, 
5,012,522, 5,309,228 are again referenced here as relevant to 
the identification of faces using pattern recognition tech- 
niques. 

Prior art employing fuzzy logic for the combination of 
information is now listed. TTiis corresponds to topic area 5 
in the above list of methods. 

Fuzzy integrals are a means of combining information 
which takes the importance of the sources of information 
into account. They have found application in many areas 
because they are easy to compute although they do require 
a training stage in which the relative importance of each of 
the sources is determined. Two types of fuzzy integral are 
described in references by Choquet and by Sugeno. The 
Choquel integral is defined by Choquet in "Theory of 
Capacities" (Annales de ITnstirut Fourier 5, pp 131-295, 
1953) and the Sugeno Integral by Sugeno in "Theory of 
fuzzy integrals and its applications" (Ph.D. Thesis, Tokyo 
Institute of Technology, 1974). 

Suppose we have a set of formation sources X={xj, 
x 2» • • • > x m}- Each of these sources provides a measurement 
of some phenomenon. We will denote the measurements by 
h(xj for source x £ . Denote by 



the set of the last m-i+1 sources in X and similarly 

the set of the last m-i+1 sources in X in which the indices 
of the elements of X have been permuted so that h(x 4 )^h 
(x 2 )^ . . . h(x m ). (In other words, is the set of the last 
m-i+1 sources when the sources are ordered in increasing 
size of measurement.) Denote the maximum and minimum 
operations by V and A respectively. The Sugeno integral of 
a function h with respect to a fuzzy measure g over the set 
of sources X is defined by 



Kx 2 ), . . . ,h(**d) - V A(*tt)A^(o)) 



(3) 



where h:X-[0,l]. The Choquet integral for a function h with 
respect to g is given by 



(4) 



where this time h:X-*9L (To satisfy the formalism it is 
necessary to define h(x o )=0.) Determination of the fuzzy 
measure g is carried out by training on test data. The fuzzy 
measure g is a set function whose value gives a measure of 
importance to the set of sources given as its argument. 

The invention will use the g 80 -fuzzy measure of Sugeno. 
For m information sources, m coefficients giving the reli- 
ability of each source are required. Fuzzy densities g* are 
defined by g'og({x.}) and are a measure of source reliabili- 
ties. Using the properties of the g x -fuzzy measure, the other 
2 m -m coefficients of the fuzzy measure are then automati- 
cally determined. 

A fuzzy measure g defined on X is a set function such that 
g(A)-»[0,l], with A some subset of X, which is defined to 
have the following properties. 



ACB->giA)Zg(B) 



(5) 
(6) 
(7) 



A g x -fuzzy measure is a fuzzy measure, which obeys the 
following condition. 

g<AVB)-g{Ay 8 (B}+\g(A)g(B), X>-1 (8) 

5 where A, B are members of the power set of X, the set of 
sources. 

If X«0, g reduces to a probability measure and obeys the 
following equation. 



10 



2 *<{**}) -1 



(9) 



For all other cases, since g(X)=l by definition, we can 
determine X by solving the equation 



n (i+ w\4i 



(10) 



which is found by repeated application of formula (8) to the 

20 set of fuzzy densities. Expanding this expression always 
results in a m-1 order polynomial in X. In other words, 
determination of k involves, in general, the solution of an 
m-1 order polynomial equation. Such an equation has m-1 
roots. However, we can use the following result to determine 

25 the unique non-zero X in the required interval. It can be 
shown that for a fixed set of fuzzy densities {g*}, g' e (0,1), 
Vi, there exists a unique X € (-1, c°), X*0 which satisfies 
equation (10). It can be further shown that equation (10) is 
positive, increasing and concave up in the interval (-1, oo), 

30 and that the line X+l crosses It at zero and exactly one other 
place. Thus, to find all coefficients characterizing a fuzzy 
measure g given its fuzzy densities g\ one simply solves 
equation (10) for the unique non-zero root in the interval 
(-1, oo) to determine X. Once X is known, all of the 

35 coefficients of the fuzzy measure g can be calculated from 
formula (8). 

Uses of the Sugeno integral for information fusion in 
vision systems are described by Tahani and Keller in "Infor- 
mation Fusion in Computer Vision Using the Fuzzy Inte- 

40 gral" (IEEE Transactions on Systems, Man, and 
Cybernetics, volume 20, number 3, pp 733-741, 1990). 
Details of the calculation of fuzzy integrals and the result by 
which the Sugeno fuzzy density measure g is calculated 
from fuzzy densities are also listed ;in this reference. The 

45 means of calculation of the fuzzy densities for each of the 
sources will be discussed in the section which describes the 
invention. Other references concerning the use of fuzzy 
integration in vision systems are those by Keiler, Krish- 
napuram and Hobson, "Information Fusion via fiizzy logic 

50 in automatic target recognition", Proceedings of the SPIE, 
volume 1623, pp 203-208, 1992), and by Keller, Qiu and 
Tahani, "The Fuzzy Integral and Image Segmentation". 
(Proceedings NAFIPS-86, New Orleans, pp 324-338, June 
1986). Characterizations of various fuzzy integrals are given 

55 by Grabisch in "Characterization of fuzzy integrals viewed 
as aggregation operators" (3rd IEEE Congress on Fuzzy Sets 
and Systems, Orlando, June 1994) and a list of industrial 
applications of fuzzy integrals are also given by Grabisch in 
"Fuzzy integral in multicriteria decision making" (Fuzzy 

60 Sets and Systems, Special Issue on IFSA Congress, 1994). 
A technique for training fuzzy integrals is given by Yoneda, 
Fukami, and Grabisch in "Interactive Determination of a 
Utility Function Represented as a Fuzzy Integral" 
(Information Sciences, volume 71, pp 43-64, 1993). 

65 The conventional algorithm closest to that of the inven- 
tion will now be explained. This algorithm is used to 
recognize faces at a single resolution. 
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The feature used in the face recognition process is that mask on the input image, the product of the image intensities 

first described by Otsu and Kurita in "A new scheme for for each of the black squares is calculated and the sum of 

practical flexible and intelligent vision systems" these intensity products is then normalized so that all 

(Proceedings of the IAPR Workshop on Computer Vision, possible characterizations lie between zero and one. Thus 

pp 431-435, 1988). A more detailed description of the 5 mc or d cr local autocorrelation coefficient is the 

features and of their use in face recognition is given by normalized mean of the intensity values of the image. The 

Kurita, Otsu and Sato in "A Face Recognition Method Using firet and SGCOQ ^ order coefficients give information about the 

Higher Order Local Autocorrelation and Multivariate Analy- oriented correlations between pixels, 
sis" (Proceedings of the 11th IAPR International Conference . , , . _.. , 

on Pattern Recognition, pp 213-216, 1991). The technique 10 . More F^isely, suppose f iy is an 8 bit grayscale discrete 
of local autocorrelation coefficients is also the subject of two of dimension I=J. The autocorrelation kernels, k, may 

patents by Otsu. The patents are Japanese patents Heisei2- be defined as » te of °( K ) couples 
101586 (Otsu, 1990) and Heisei2-101591 (Otsu, 1990). This 

scheme has since been expanded and evaluated by Goudail, (i*d„)-{-ifiA} m {-i> Q >i} 
Lange, Iwamoto, Kyuma and Otsu in "Real Time Face 15 

Recognition Using High Order Autocorrelations" where 0(k) e{0,l, 2} is the order of the autocorrelation and 

(Proceedings of the IEICE Spring Meeting, page labeled each couple represents an offset from the central pixel. For 

7-277, 1993, and in a second reference by the same authors example, using the representation in which the y-axis of the 

"Fast face recognition method using high order auto- image is directed towards the bottom of the page and the 

correlations" (Proceedings of the International Joint Con- 20 x-axis towards the right, the second order kernel which is 

ference on Neural Networks, Nagoya, pp 1297-1300, 1993). pictorially represented by a diagonal from top-left to 

The process followed in the system previously described bottom-right would be represented by {(-1, -1), (0,0), 

is shown schematically in FIG. la. In the training stage, (1,1)}. For each of these kernels, we calculate the 0(K)th 

given a series of gray -scale input images, each image is orc j er autocorrelation coefficients using 
characterized using local autocorrelation coefficients to give 25 

a feature vector for each image. The inter- and intra-class ^ Jt4 0 (k) (12) 

covariance matrices of the feature vectors for the known Ck ~xh) ' - 2 n f M >&» 

training images are used to generate a feature mapping K tml /_2 «=o 

which maximizes the difference in feature vectors between . , . a „„ • „ nKtant ■ n K „ 

, , , , , v.M .... , where N[K) is a normalizing constant given by 

each of the known classes (persons) while minimizing the 30 

differences between the features of faces of the same class N ^ l (13) 

(person). The analysis used to calculate this mapping is (/ - 2)^ - 2) (2 & - iy<*)**- 

. called linear discriminant analysis. The Euclidean distances 

between the centers of the clusters of mapped features for where N(k) is a normalizing constant given by 
each class and the features for images belonging to the class 35 

are minimized. This principle is utilized in the recognition #(K)=i/(/-2)(/-2)(2 8 -iy Jt ' c) - 1 ( 13 ) 

stage for the test images. 

In FIG. lb, the mapping found by linear discriminant and the coefficients C K lie in the range [0,1]. Since there are 

analysis is applied to the training data to calculate the values one, four and twenty of the zeroth, first and second order 

of the vectors after the mapping transformation. The vectors 40 coefficients respectively, concatenation produces a 25 ele- 

for each class, that is the vectors which correspond to images ment feature vector which is used to describe each image, 

from each face or person (or object if the method is not being The factor of 2 8 -l corresponds to the use of an 8-bit 

used for face recognition), are averaged resulting in one gray-scale representation. 

mean vector for each known person (or object). This mean method of u - near discriminant analysis (or multiple 

vector is referred to as the cluster center. It is an indicator of 45 discriminant analysis) is described by Duda and Hart in 

the center of the region in which mapped vectors corre- «p at tern Classification and Scene Analysis" (Wiley 

sponding to a known face are likely to be present. lnterscience, pp 118-121, 1973.) 

In FIG. lc, the procedure for recognizing test images, the . , _ , . . , r , , . . 

... • i_ n i * .u ■ ? ? ■ \u The mter-class scatter matrix is defined to be 

test stage, is shown. By applying the mapping found in the 

training stage, the features for the unknown images are 50 c ^ 

transformed in such a way that the distances between the s*- ? St 

transformed feature vectors for faces which were in the 1-1 
training set and the cluster centers for these faces is mini- where 
mized. Thus recognition can be performed by thresholding 

the distance measurement between an unknown face and all 55 Si- X (x - m,) (* - m,) 7 * 
of the cluster centers. The closest cluster center is the most x * Cl 

likely candidate for the identity of the unknown face: if the where T denotes the transpose operation (to form the outer 

distance exceeds some threshold, then the test face can be product); the vectors x are the feature vectors; Q are the c 

rejected as being unknown. classes, each of which has n,. members; and m ( . are the mean 

We now describe this procedure in more detail. 60 vectors for each class. That is m,- are the cluster centers given 

The autocorrelation coefficients are calculated by scan- by lne following equation, 
ning the set of masks shown in FIG. 2 over the gray-scale 

image to be classified. The areas which are represented by j_ (is) 

black squares represent pixels whose mask has a numeric nti " n- t xeC{ 

value of one whereas those represented by white squares 65 

have a mask of zero and are not used in the calculation for The intra-class scatter matrix is defined by the following 

that coefficient. For each of the possible positions of the equation. 
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In another aspect of the present invention, there is pro- 
Si, - I n ( (m, - m) (m, - m) T ^ vWcd a process for solving a problem of recognition of faces 
M ' or other images. According to this process, captured images 

f4 . . » . , „ , . A . 4 _ . of persons to be identified are read, and then characterized 

It is shown by Duda and Hart that the transpose of the usi mc mcthod tcrfficd { x autocorrclalion cocfficicnts , 

matrix whose columns are the eigenvectors c, of the gener- Tqc charactcrization rcsuIts in a vcctor of numbcre which 

auzed eigen-proolem characterize the image. By shrinking (or expanding) the 

SflerXS^c, (18) image different characterizations may be calculated, one for 

gives the optimal mapping. This mapping is the one used in cach resolution. These characterizations are then mapped or 

the invention. 10 transformed into new vectors, this transformation corre- 

Previous descriptions of the invention and related material sponding to the mapping of the feature vectors from a 

have appeared in the following two publications. Their feature space, a coordinate system, to a recognition space, a 

publication date is less than one year before the submission possibly different coordinate system. By comparing the 

of this patent. Arbuckle, Lange, Goudail, Iwamoto, locations of the mapped vectors in the recognition space 

Takahashi, Ishii, Mori, Kyuma, aid Otsu in "Face Recogni- 15 with the transformed averaged features for images of known 

tion System Incorporating Classifier Combination by Fuzzy faccs at thc sarnc resolution as the image to be tested, a series 

Integral" (Proceedings of the 3rd International Conference of dista °ccs can be calculated, each distance measuring the 

on Fuzzy Logic, Neural Nets and Soft Computing, IIZUKA, distance between the mapped or transformed point and the 

pp 393-394, August 1994) describe a system very similar to transformed cluster centers, the mapped averages of the 

the invention. Arbuckle, Lange, Iwamoto, Otsu and Kyuma 20 feature vectors for each class of image in a training set. 

in "Multi-resolution Face Recognition System Incorporating Taking these sets of distances for each image but calculated 

Fuzzy Integration" (Proceedings of the IEICE General from the characterizations at many resolutions, the proce- 

Conference, Fukuoka, Systems Volume 2, pp 252, March dure of fuzzv integration is applied to yield a further set of 

1995) describe the invention. differences, or alternatively a set of scores, whose value can 

An internal real world computing partnership memo on 25 be used in a similar way to distances to characterize the 

this topic by Iwamoto, Goudail, Lange, Arbuckle, Kyuma, similarity of the image to known images. These distances or 

Otsu "Classifier System for Visual Images" (Real World scorcs arc mcn classified by picking the minimum distance 

Computing Technical Report TR -94001) whose internal ( or P°ssiMy maximum score or the most suitable value of 

distribution date was more than one year before the submis- some faction °f these distances or scores) in order to 

sion of this patent but which was not presented for general 30 determine the most likely identity of the person to be 

publication, contains a description of an invention very recognized and also in order to determine if the image (and 

similar to the one presented. its series of similar images of the same scene or person but 

at different resolutions) is in fact of a person or object not 

• SUMMARY OF THE INVENTION previously made known to the system, thus permitting both 

Accordingly, it is an object of the present invention to 35 the recognition of known faces or objects and the rejection 

eliminate the above -described problems in the related art of unknown faces or objects. 

and to provide a novel and improved method of recognizing In a third aspect of the present invention, there is provided 

faces or other images. a process for realizing a statistical procedure together with 

To achieve this object, in one aspect of the present a means of combining the results of this statistical procedure 

invention, there is provided a procedure for solving the 40 whose method of operation is fuzzy integration. The system 

problem of the recognition of faces or other images and a is provided with three sets of images of faces. For each 

means of combining the results of the procedures. When image, corresponding images are calculated at differing 

presented with images of known persons (at least two), the resolutions or corresponding images at different resolutions 

number of images being sufficient to permit the application are presented from some means of capturing images and 

of the method termed linear discriminant analysis, these 45 digitizing them. The first of these sets of images comprises 

images are analyzed using a technique termed local auto- images of the persons who well be assumed known by the 

correlation coefficient feature analysis or, simply, local auto- system. The second set of these images comprises different 

correlation coefficients, this analysis resulting in a series of images of all of the persons in the first set together with 

feature vectors, one feature vector for each image, which is images of persons not known to the system, the classification 

then analyzed using the procedure termed linear discrimi- 50 of known and unknown images also being known to the 

nant analysis. The results of this analysis run on versions of system in advance. The third set of these faces comprises a 

the same image at multiple resolutions being combined by mixture of yet more images of persons known to the system, 

the technique called fuzzy integration in the forms termed that is persons whose images are in the first training set of 

Sugeno Integration or Choquet integration although other images, and images of persons who are not known to the 

forms of integration may also be suitable for this purpose, 55 system, not necessarily the same unknown persons who 

the integration being with respect to a fuzzy measure, fuzzy were present in the second set of images, 

measures including but not limited to the Sugeno measure. For the first of these sets of images, all of the images are 

The fuzzy measure is itself generated by using a second classified using the method of local autocorrelation coefiB- 

training set of training images. The combined characteriza- cients resulting in a classifying vector which describes each 

tion may then be classified using a method called least 60 image at each resolution. At each resolution, applying the 

distance classification in which the Euclidean distance of the method of linear discriminant analysis (or multiple discrimi- 

mapped point from the points corresponding to the mapped oant analysis as it is sometimes called) to the sets of images 

average of the points for each person's face in the training results in the calculation of a transformation matix, not 

date is calculated, the closest being taken as the most likely necessarily a square matrix if the dimension of the charac- 

recognition result, rejection of unknown persons being 65 terizing vector is to be reduced in the transformation, which 

implemented using thresholding of these distances or some has the property of grouping the points belonging to each 

other means of similar effect. face (or class) more closely together, whilst, at the same 
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time, maximizing the spread in the groupings or clusters 
which represent these known faces. Applying these map- 
pings to the features for the training set at each resolution, 
the correct mapping matrix being used results in mapped 
feature vectors for each resolution. The cluster centers, that 5 
is the average of the mapped points for each of the images 
for each particular class or known face can then be calcu- 
lated under that mapping. These cluster centers and this 
mapping are then used in the subsequent stages. 

For the second of these sets of images, the performance of 10 
the system at each resolution is tested by classifying the new 
images using the technique called local autocorrelation 
coefficients, applying the corresponding mapping for that 
resolution and calculating the distances between the mapped 
feature and the cluster centers calculated in the previous 1S 
stage. By employing the distances or some measures derived 
from these distances in conjunction with some thresholding 
technique, the percentage recognition rates and rejection 
rates for the system as trained on the first set of faces can be 
gauged by this test on the second set of faces. It is the 2 o 
performance of the system at each resolution which is used 
to generate the fuzzy densities used in combining the results 
from multiple resolutions. For each resolution and for each 
face, the recognition rate of the recognition system corrected 
for an error rate is used as the fuzzy density corresponding 2 s 
to the quality of the classifier at that resolution and on that 
known face. 

Finally, for the third set of images, the images are again 
classified using the method of local autocorrelation coeffi- 
cients and the mapping matrix derived in conjunction with 30 
the first set of images is used to transform the vectors into 
the mapped vectors in the usual way. The distances between 
the mapped image feature vectors and the cluster center are 
calculated as when used in conjunction with the second set 
of images. At each resolution, the distances between the 35 
mapped features for the face and between the cluster centers 
are calculated. These distances are then converted into 
scores by using a formula and these scores are combined 
using fuzzy integration of, for example, either the Sugeno or 
Choquet type, the fuzzy measure required for the calculation 40 
of the integrals coming from the percentage recognition 
rates corrected for an error rate and the formulas describing 
the Sugeno fuzzy density. This results in a single measure- 
ment for each image between it and between points repre- 
senting the combined centers of the classes at the multiple 45 
resolutions. This measurement can then be used for recog- 
nition in the manner described previously, namely that some 
form of thresholding may be performed on some combina- 
tion or function of the distances or scores or measures 
corresponding to each of the images to reject unknown 50 
faces, the optimal measurement corresponding to the known 
face. 

In a fourth aspect of the present invention, there is 
provided a process for realizing a set of neural networks 
together with a means of combining their results whose 55 
method of operation is fuzzy integration. Given three sets of 
images, two for training and one for testing in which there 
is an overlap of faces in all three sets corresponding to face 
which are to be known to the system, once these faces at 
multiple resolutions have been classified by using local 60 
autocorrelation coefficients, or possibly using a neural net- 
work implementing characterization using local autocorre- 
lation coefficients, the following procedure analogous to the 
previous ones can be used. For the first set of images, a 
neural network can be taught to perform the mapping 65 
previously carried out by the process and the application of 
the statistical procedure. Once the neural network has 



learned this mapping, the procedure can proceed as before. 
The mapping is applied to find the cluster centers; the 
mapping is applied to the second data set and the recognition 
and an error rate are determined; the measurements of 
recognition performance and of an error are combined to 
form the fuzzy measure, the fuzzy measure is used to 
combine the measurements produced at multiple resolutions 
and the combined results are then examined by some func- 
tion which returns the optimal recognition result from the 
data given, and which applies some form of thresholding to 
permit the rejection of unknown faces. It is possible that a 
single neural network can be trained to both find the local 
autocorrelation coefficients and to perform the mapping at 
the same time. 

Several additional objects and advantages of the present 
invention are: 

a) to actually recognize facial images (or images of other 
classes of objects) as opposed to locating them in an 
image or performing some other form of preprocessing 
helpful in recognition; 

b) to recognize facial images reliably when used with 
large data sets having a large number of classes 
(persons or classes of objects), as can be shown by 
statistical analysis of recognition and rejection rates of 
the present invention using images of over one hundred 
persons; 

c) to perform recognition without the requirement of 
multiple sequential images for face localization within 
an image, although multiple images of faces are still 
required; 

d) to perform recognition without the requirement of 
specialized images of irises or other body parts and to 
be able to perform the recognition from images of 
heads and upper torsos only; 

e) to perform recognition without necessarily requiring 
the long training times required for neural network 
techniques; 

f) to perform recognition without the requirement of using 
color images although color images could be used; 

g) to use multiple resolutions concurrently (so far as is 
possible) in the recognition process rather than incre- 
mentally; 

h) to perform recognition without the requirements that 
the facial images to be recognized are precisely aligned 
in some particular way; 

i) to perform recognition without any requirement for the 
use of three dimensional information; 

j) to perform recognition using multiple resolutions rather 

than one single resolution; 
k) to perform recognition in a deterministic manner in that 
the results of tests of recognition performance using the 
same data return identical results 
1) to combine information from multiple resolutions in a 
manner which takes account of the importance and 
reliability of the sources of information used. 
Further objects and advantages are to be able to imple- 
ment the invention using statistical or neural network 
methods, to be able to carry out recognition using fewer 
training images than required for traditional neural 
approaches, to be able to achieve more accurate recognition 
performance by combining information from multiple reso- 
lutions in a manner more conducive to better performance 
than simple averaging, to be able to use different means of 
calculating the fuzzy densities (which are similar to weight- 
ings given to the information sources) during the combina- 
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lion of information and to be able to perform rejection of sake of clarity and to further the explanation, it is assumed 

faces (or other objects) that are unknown as well as recog- here that the images are produced by pixel averaging, each 

nize faces (or other objects) that are known. averaging operation reducing the size of the image. Thus, if 

The above and other objects, features and advantages of blocks of two pixels by two pixels are averaged all over the 

the present invention will become clear from the following 5 j raagc anc j a DCW i mage ma d c of these averaged results, the 

description of the preferred embodiment thereof, taken in new image wi ji ^ Da jf the s i ze 0 f lnc i^ge Approximately 

conjunction with the accompanying drawings. tf , he - mz& breadth or height have an odd number of pixels) 

BRIEF DESCRIPTION OF THE DRAWINGS If lheavera S e » Jf*™*. ove ' lhree b V * re ? blocks, the 

new image will be approximately one third of the original 

FIG. la is a schematic diagram of the processes in the 10 size. 

closest related system during the training stage in which the A1 . . . , , 

, c *• * • TiTj Alternatively, when averaging over two by two blocks, 

mapping or transformation matrix is calculated. /' , . f. , « , , . 

JLv ~ . , ... „ , . the block itself can be replaced by a block the same size 

HO. lb is a schematic diagram ot me processes in the fc . , he avef va , ue 

in which case the imaee size will 

closest related system during the training stage in which the 1C 4 l • i i_ < ,c • r . * c iL * -n 

. . . * .i_ i r I i . a c. 15 not shrink but the information content of the image will 

cluster centers of the known faces are calculated after , T _ - , 

,- ,. nf( , «™„'„,„i. f ; decrease. If averaging is performed over a block with a size 

application of the previously calculated mapping. „ , . . t , . , 

" _ . , . r . • . of the minimum of the height and width of the image, the 

FIG. lc is a schemata diagram of the processes in the new . wiu ^ an . which is com ^ of 

closest related system durmg the testing stage for two faces, ^ information around some 

one known and the other unknown. 20 r ,u j c.u • w u u . u ,*u 

of the edges of the image which would not be covered by a 

FIG. 2 is a diagram of a the local autocorrelation kernels. ^ 0 fb\«*s of me relevaQt size {s discarded although this 

Black squares represent mask areas of value 1; white areas ^ QQ{ {Q bc ^ ^ Qec 0f { m objcct is 

represent mask areas of value 0. simply (hat images of predsdy ^ ^ subject maUcr ^ 

FIG. 3a is a schema tic -diagram which gives an overview at different resolutions are produced, the exact method of 

of the invention as applied to the recognition of an unknown 25 ^ produclion QOt being imporlanl t0 the operation of the 

P ers0D * invention. 

FIG. 3fc is a schematic-diagram which gives a more 

detailed view of the invention during the processing of an Id the 116x1 sla S e of lne method, for each of the resolutions 

image of an unknown face. °f tne image, the local autocorrelation feature vector is 

FIG. 3c is a schematic^iagram which gives a detailed 3 ° calculated as described in the section 25 describing the prior 

view of the process of fuzzy integration. art Th* means that » m the case of * sav * five resolutions, five 

FIG. 3d is a schematic-diagram which shows two areas f m< f autocorrelation vectors will be generated which 

where neural networks may be applied to transform the data. descnbc the unknown ima S c at cach °/ ' he five rcsoluUons. 

M . a . . c c i % u • « This also limits the minimum size of the image since the 

FIG. 4a is a flowchart of a face-recognition algorithm in 35 . . . ° . . 

... t c t . . . f. . . .. c * image must have a size of at least three pixels by three pixels 

rt pmrwlimpnt r»f thp. nrp.c^.nt invpntinn nunntx its first ° r j r 



in order to apply the local autocorrelation method. 



part embodiment of the present invention during its first 
training stage. 

FIG. 46 is a flow-chart of the face-recognition algorithm In the next stage of the method, each of the local auto- 

in part embodiment of the present invention during its correlation feature vectors is transformed by multiplication 
second training stage. 40 with one of a set of mapping matrices which have been 

FIG. 4c is a flow-chart of the face-recognition algorithm calculated by the method of linear discriminant analysis 

in part embodiment of the present invention during its (sometimes called multiple discriminant analysis) and with 

testing stage, respect to a training set of known faces as will subsequently 

, be described. Thus, in the case of five resolutions of the same 

DESCRIPTION OF THE PREFERRED « therc are five mapping ^ ^ beeD 

EMBODIMENT generated from training data at the same resolution. Appli- 

An embodiment of the present invention will be described cation of the mapping matrices has the effect of transforming 

in detail while referring to the accompanying drawings. the feature vectors into new feature vectors, not necessarily 

FIG. 3a is a flow chart which gives an overview of the 50 of the same size, 

present invention. The invention is shown processing one ^ m y ealioa has stored a set of vectors for each reso- 

unknown image, liie process tor attempting to recognize . ... , 4| _ . , . . . . . 

. & . • • 1 4 1 .l lution which are called the cluster centers and which repre- 

many unknown images is simply to apply the same process , . f f , r . . - r _ 

' . r r™ . • j u j **u sent the averaged features for each of the known faces after 

to each image in turn. J ne invention is described with . ... . <• , . 

reference to FIG 3a transformation by the mapping matrix for that resolution. 

„ . . . ... . . . TTiis allows a series of distances to be calculated between the 

The invention employs images at multiple resolutions. , t . . c , , 

™ i. r 1 j mapped vector corresponding to the unknown face card and 

The number of resolutions used must be greater than one. t r f4 , . . r t . . . c ... 

. . r. . • , each of the cluster centers. The distance used is the Euclid- 
Each of the images can be generated from one original ,. # . . . . , e 4 , „ c 

to , , *? . , . 5 1 . . ean distance which is the square root 01 the sums ot the 

image by scaling the image using pixel averaging to shnnk Ct , . , 4 c u 1 

. ° ' . tl • a % i.- 1 squares of the differences in the elements for each element 

the image or may be obtained from multiple cameras or 60 . t , . uu . , 

4 . . 0 / . ™ . c t in a vector and a second vector although some other distance 

other image capturing means. The number of resolutions , , 

. . & r ..... . c . . . t measure could be used. 

used has no theoretical Limit if many machines are used to 

capture the images. In the case where the images are Thus, for each resolution and for each of the cluster 

obtained by averaging of pixels, the number of resolutions centers at that resolution, a distance between the mapped 
may be limited by the image size if averaging of pixels 65 feature vector and each of the cluster centers can be calcu- 

produces smaller images or by the fact that averaging will lated. For each esolution, there will be as many distances as 

eventually produce an image of only a single color. For the there are known faces. We use the formula 
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image at the five resolutions, the distance vectors that would 

1/4* (19) usually be thresholded theu being manipulated and subse- 

P quently combined using fuzzy integration to result in com- 

^ 3/rf * pounded or combined results whose values can then be 

5 searched for the optimum recognition value by some func- 

suggesled by Xu, Krzyzak, Suen in "Methods of combining tion such as minimum or maximum and which can then be 

multiple classifiers and their applications to Handwriting thresholded in the usual way to achieve the recognition of 

Classification" (IEEE Transactions on Systems, Man and known faces and rejection of unknown faces. 

Cybernetics, volume 22, pp 419-435, 1992) to convert the We will now describe each of the components given in 

distances d into measurements h in the range [0,1] for FIG. 3a in more detail. 

resolution k given p distances d indexed by i. This means 1 The first element in the diagram is feature detection. This 
that the distances for each resolution are transformed into a operates as described in the section on prior art. The masks 
series of characterizing numbers between zero and one in shown in FIG. 2 are scanned over the image. Wherever there 
which the larger the number is, the more likely the identity is a black square, the gray value under that black square is 
of the person in the unknown image corresponds to the multiplied with the other values under the black squares in 
known face corresponding to the largest value. The above 15 that mask, the mask being scanned over the image and the 
formula is simply a means of transforming distances which values produced being summed and then divided by the 
range from zero to infinity to numbers in the range zero to normalizing factor, this process taking place for each of the 
one which can be used for fuzzy integration using Sugeno twenty-five masks shown and this procedure being as 
integrals. The use of this formula is not meant to be described in equations (11), (12) and (13). This procedure is 
restrictive or compulsory and the requirement that the values 20 entirely as described in prior art. It is pointed out here that 
lie between zero and one does not apply to some kinds of there may be some advantage in using what is called kernel 
fuzzy integration such as Choquet integration. dilation by Goudail, Unge, Iwamoto, Kyuma and Otsu in 
The sets of numbers are now combined using fuzzy "Fast face recognition method using high order auto- 
integration of Sugeno or Choquet or some other type with correlations" (Proceedings of the International Joint Con- 
respect to a fuzzy measure as will be subsequently 25 ference on Neural Networks, Nagoya,pp 1297-1300, 1993). 
described. For example, in the case where there are five sets This refers to using the same kernels as shown in the 
of distances corresponding to five resolutions, there will be diagram but spaced over blocks of five by five, seven by 
one number in each set which corresponds to each of the seven, nine by nine, and so on. In the five by five case, for 
known faces. Thus, if there are, say, sixty known faces, there example, the mask which looks like a diagonal of a matrix 
will be sixty numbers in each set. Each of these numbers will 30 in which the topleft, center and bottom-right squares are 
lie in the range zero to one and the size of the number varies black and the rest white would, in the five by five case, have 
as the similarity of the unknown image to the known face the top-left, center and bottom -right squares black. This can 
which gave rise to that number. For each known face in rum, be thought of as multiplying the couples given in equation 
the numbers corresponding to that known face are combined (5) by a whole number greater than or equal to one. 
using fuzzy integration to reduce the five sets to one set of 35 Corresponding charges need to be made to the scaling value, 
combined numbers in which, again, the larger a number is, Since this is not 'necessarily part of the invention, its use is 
the more the unknown face resembles the known face that merely mentioned here. 

gave rise to the number. In other words, if there are five The next element in diagram 3a is that of a matrix 

resolutions, the numbers for each face are combined to transformation generated by linear discriminant analysis, 

produce a new number. Thus, the total number of classifying 40 The process of calculating the optimal mapping matrix from 

numbers would be reduced by five in the case of five training data has already been explained in the section 

resolutions. This will subsequently be described in more describing the prior art. It consists of the calculation of the 

detail. inter- and intra-class scattering matrices for the classes 

Having a single set of numbers, one number for each of which correspond to all of the known faces. Each known 

the known faces, the actual recognition or rejection stages 45 face corresponds to a class and there must be at least two 

are now performed on this sequence of numbers. Either the classes. Limitations in the method mean that with an insuf- 

largest number can be taken as the most likely face or the ficient number of samples, the correct matrix calculation 

reciprocals of the numbers can be used in which case the cannot be performed. This is a well known problem with this 

smallest of these (reciprocal) numbers will give the best method and it is mentioned in passing for the sake of 

matching face or some function of the numbers can be used 50 completeness. 

which will give a measure of the most likely face. For the This calculation, together with the characterization by 

case of rejection, thresholding of the numbers or of the local autocorrelation coefficients are performed exactly as 

reciprocals of the numbers or of the function which is described in the prior art. The result of these calculations is 

applied to the numbers for recognition can be used for a matrix which is used to map feature vectors at a particular 

rejection of unknown faces as will subsequently be 55 resolution together with a set of mapped feature vectors 

described in more detail. corresponding to the averaged feature vectors for each class. 

In overview then, the invention corresponds to the use of Again, this is exactly as described in the prior art save that 

multiple recognition procedures, as described in prior art, the distance measurements are the final product of this 

but incorporating a combination step using fuzzy integrals, section rather than the intermediate value which is then used 

and an associated training procedure therefor, to produce a 60 in the recognition and rejection procedures, 

single recognition procedure which carries out recognition The next element in diagram 3a is fuzzy combination of 

and rejection with the benefit of information from multiple the distances from each resolution to form a set of combined 

resolutions. distances for each known face. As has already been 

For example, in the case o" five resolutions, this corre- described, at this stage, for each unknown image, the result 

sponds to five of the recognition procedures comprised of 65 of previous processing is several sets of distances, one set 

local autocorrelation coefficients and linear (or multiple) corresponding to one resolution and each distance corre- 

discriminant analysis being run in parallel on the same sponding to the distance between the unknown face and the 
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faces in (he set of known faces at that resolution. The 
distances are transformed into characterizing numbers using 
formula (19). In the case of these numbers, the larger the 
number is, the closer the face corresponding to the number 
is to the unknown face. Thus the position is as shown in FIG. s 
3b. In this diagram, each of these sets of numbers are shown 
as vectors in which the first element of each vector corre- 
sponds so the first known face, the second element of each 
vector corresponds to the second known face and so on, 
there being, in this case, four known faces. Elements are io 
taken from the five vectors as shown and combined using 
fuzzy integra lion of Sugeno or Choquet type to form a new 
single vector. The use of Sugeno or Choquet integrals is not 
meant to be restrictive or compulsory. The fuzzy measure 
used in the calculation is calculated from additional training is 
data as will be subsequently described. Once the combina- 
tion is performed, the resulting vector may be treated in a 
similar manner to the single vectors of distances produced 
by each of the resolutions in the case described in the prior 
art. 20 

The final procedure in diagram 3a is described as distance 
classification. If the reciprocals of the characterizing num- 
bers are taken, the largest number in the set of characterizing 
numbers will become the smallest number. Thus the smallest 
reciprocal number is taken as the best match. If the recip- 25 
rocals of the numbers are not taken, the largest number is 
taken as the best match. Better results can be obtained, by 
using a contrast measure 



' di+d 2 



(20) 30 



where d 2 and d 2 are the shortest and the second shortest 
distances (or reciprocal classifying numbers) respectively as 
measured in the recognition space as shown by Goudail, 
Lange, Iwamoto, Kyuma and Otsu in "Fast face recognition 35 
method using high order auto-correlations" (Proceedings of 
the International Joint Conference on Neural Networks, 
Nagoya, pp 1297-1300, 1993). If the classifying numbers 
themselves are used, the formula needs to be modified 
appropriately to give a positive number by using the fol- 40 
lowing formula: 



di +d 2 



(21) 



45 



Thresholding of this contrast measure rather than the recip- 
rocals of the classifying numbers or of the classifying 
numbers themselves can be used for recognition, the advan- 
tage being that if the two closest cluster centers are at similar 
distances, or if the combined classifying numbers for the two 50 
most similar faces are of a similar size, then the contrast 
value given by the correctly applied formula chosen from 
(20) and (21) will help eliminate the case where no one 
match stands out from the others. 

An overall view of the invention has now been given in 55 
which a description of several of its parts has been given. 
There remain several points which require further explana- 
tion before the method of operation is explained. These 
points are: 

a) the derivation of the fuzzy measure from training data; 60 

b) the method of fuzzy integration itself. 

We will now go on to describe each of these points in turn. 

The fuzzy measure is derived from training data. The 
invention requires three sets of images of faces in order to 
operate. The first set of images is the set of images of known 65 
faces. For each resolution, there is a local autocorrelation 
feature extraction step followed by a linear discriminant 



analysis step which uses the features calculated at that 
resolution. This can be thought of as multiple copies of the 
recognition system described in the prior art being run in 
parallel, each at a different resolution, each resolution look- 
ing at the same image but in a different amount of detail. 
Once the training has been performed on the first set to 
obtain the mapping matrix, the mapped cluster centers are 
also obtained by applying the mapping matrix to the aver- 
aged features corresponding to the images for each face. 

A second set of images is then required to train the fuzzy 
combination procedure. This set needs to have different 
images of faces in the first set as well as images not 
previously seen by the system. The set of images is shown 
to the sets of recognition systems as described in the prior art 
to collect data on the performance of each of the classifi- 
cation systems at each resolution and for each face. 

We define the recognition rate for a classification system 
or for a classification system with respect to a particular face 
as the percentage of known faces which are correctly rec- 
ognized. We define the false access rate as the percentage of 
unknown faces which are falsely claimed to be recognized 
by a system. We do not use data on the number of known 
faces which are incorrectly recognized as another different 
known face. Denoting the recognition rate as rec and the 
false access rate as far, we ran define the formula 



(22) 



as a means of calculating (raw) fuzzy densities g,y where one 
of the indices refers to the known face and the other to the 
classifier (which can also be thought of as the resolution) for 
each face and for each classifier Equation (22) states that the 
fuzzy density for a particular resolution and for a particular 
known face is given by one minus the square root of half of 
the sum of the squares of the false access rate for the face at 
that resolution and the square of one minus the recognition 
rate for that face at that resolution. The recognition rate is the 
percentage of the images of known faces at a particular 
resolution that were correctly recognized and the false 
access rate is the percentage of unknown faces who were 
incorrectly recognized as that face. These fuzzy densities are 
only raw fuzzy densities because before they are used, they 
undergo further processing as will now be described. 

Two kinds of processing have been tried on the fuzzy 
densities giving similar results. To label them, we call them 
normalize-by- file and normalize-by-face. 

In the case normalize-by-file, we sum all of the values, 
then set all zero values to an arbitrarily chosen negligible 
value (currently 10~ 20 ), then divide each of the matrix 
values by the sum. All values are then multiplied by the 
number of known faces and by a user-chosen scaling factor. 
The negligible value is to correct for the case where at leas! 
three of the fuzzy densities for each face are zero. In this 
case, the solution method for X given by equation (10) fails. 
Moreover, if the fuzzy density for a classifier is zero, the 
classifier is not considered in the fusion or merging process. 
The user-chosen scaling value is chosen to maximize the 
performance of the system in a manner to be subsequently 
explained. 

In the alternative scaling method, normalize -by-face, for 
each of the known faces, we sum the values, perform the 
replacement of zeros as before, divide all values by the sum 
and finally multiply all values by the given scaling param- 
eter directly. The values obtained are of comparable sizes to 
those found by the normalize-by-file method. 

Other alternatives to these scaling methods exist but these 
two methods have been shown to perform better than the 
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others tested so far. Their use is not meant to be restrictive classifying numbers. That is, each of the pairs will be sorted 
or compulsory although some form of scaling is required so that the pair members which correspond to the measure- 
when the raw fuzzy densities are derived from training data ments or classifying numbers will be in increasing order of 
in the manner described. magnitude even though the elements for the fuzzy densities 

The value of the user defined scaling parameter is decided 5 will be in no particular order save for their association with 

as follows. During the testing on the third test set, the value their corresponding classifying number, 

of the scaling parameter is adjusted to give the best possible The next step that is required is the solution of equation 

recognition results as measured by the score defined by the (10) using the dizzy densities from the matrix to solve for X. 

following equation This solution permits the fuzzy measure to be calculated 

]0 using equation (8). If we denote the sources, that is the 

~~ (23) classifiers at each of the five resolutions by SI , S2, S3, S4, 

score ^ j far 2 + n - rtc? S5, and assume, for the purposes of illustration that they are 

score- 2 ordered by size of increasing measurement in the order S5, 

, t . c , . . ... 4 c iL . S3, S2, SI, S4, then only the fuzzy measures for the sets of 

where the false access rates and recogn.Uon rates refer tins {S5,S3,S2,S1,S4}, {S3,S2,S1,S4}, {S2.S1.S4}, 

hme o the performance of the system over the est data. 15 {S1 ^ ^ {S4} need be' calculated. These measures ci 

namely the data in the third test set. This optimization may ^ £ / licalion of formula (8) startin ^ ^ 

be seen by drawing a graph of score versus scaling param- {M} and w ^ backwards throug y ihc Us t. It is now 

eter and may be automatically chosen if required. po Jible to apply equations (3) and (4) to calculate the 

The method of fuzzy mtegration itself will now be rn.™.~» i- \ • \/ r»„ _ ■ lU ■ 

j u j tt_ / n • . * i_ • n^. Sugeno or Choquet integrals as required. By applying this 

described. The process of fuzzy integration is shown in Flu. nn to , • . ^ r,i_ i \ ■ ^ 

it»l j „.u i i_ cc i** 20 procedure in turn to each of the columns of the matrix with 

3c. In this diagram, the particular numbers of 5 resolutions f. , . . . - . , c - 

and 14 known faces have been used in the diagram for the ?» corresponding elements of the vectors a vector of fuzzy 

purpose of illustration and to make the explanation of the inte S rals can be calculated which may then be used in the 

method clearer. This is not meant to constitute a limitation, recognition and rejection steps as previously described. This 

and indeed the invention performs well with larger numbers. procedure is described in the reference by Tahani and Keller 

In diagram 3c, the combination of five sets of classifying 25 "Information Fusion in Computer Vision Using the Fuzzy 

numbers lying between zero and one with respect to a set of Integral" (IEEE Transactions on Systems, Man, and 

fuzzy densities is shown. There are five sets of classifying Cybernetics, 20(3) pp 733-741, 1990) and its explanation is 

numbers because, for this example, five resolutions are given here to aid the understanding of the invention, 

being used. The matrix of fuzzy densities has five rows This embodiment is applied to face recognition, but the 

corresponding to the five resolutions. Each of the vectors of 30 present invention is not restricted thereto. Its use for recog- 

classifying numbers has 14 elements and the matrix of fuzzy nition of animals or of machine parts or of vehicles has been 

densities has 14 columns corresponding to the use of 14 suggested. It is a fully general image recognition system, 

cluster centers which in turn corresponds to the use of 14 The only requirements are that there be a sufficient number 

known faces or 14 sets of known facial images. For the of images to permit the calculation of the eigenvalues and 

purposes of the illustration, and without loss of generality, it 35 eigenvectors used in the linear discriminant analysis 

is assumed that the elements in the vectors and matrix are procedure, and that these images are in sets of multiple 

ordered such that elements in corresponding columns cor- images for each individual object to be recognized, 

respond to the same known face. Thus, the elements in the The method of operation of the preferred embodiment, as 

leftmost position of the vectors of classifying distances all described in the previous section, will now be given below, 

correspond to the first unknown face as does the first column 40 Three sets of images (of faces or other objects) are used 

of the matrix of fuzzy densities. Moreover, it is assumed that during the operation of the invention. The first set, a training 

the resolutions at which the characterizing numbers and set, consists of multiple images of the faces to be learned by 

fuzzy densities were calculated correspond so that the first the system, each of these images themselves being present 

vector corresponds to the top row of the matrix of fuzzy at multiple resolutions. These multiple resolutions are fixed 

densities, the second row of the matrix of fuzzy densities 45 during this training stage and so all other images from the 

corresponds to the resolution at which the second vector of other two sets must also be present at these resolutions. As 

characterizing numbers was characterized and so on. The an example, 50 images of sixty persons at five resolutions 

method of fuzzy integration of the classifying numbers can be used to train the system. For each of the resolutions, 

corresponding to the first known face will be explained. The training consists of characterization using local autocorre- 

method of combination for all of the other 13 known faces 50 lation feature extraction, calculation of the inter- and intra- 

in this example being merely a repetition of this procedure class scattering matrices and the calculation of the optimal 

working column-wise along the data. transformation matrix. This optimal transformation matrix 

The first elements of each of the vectors of classifying (one for each resolution) is then either applied to the average 

numbers and the first column of the matrix of fuzzy densities of the features for each class, that is each known image set 

are required to integrate the information for the first known 55 or each known face, or the transformation can be applied to 

face, the output of the procedure being a single number. The all features individually and the averaging of the features to 

element from the topmost row and leftmost column of the form the cluster centers performed after the transformation, 

matrix corresponds to the leftmost entry of the first vector This is the same training scheme as described in the prior art 

and so on for each of the rows of the matrix as has already and differs only in that this training scheme is run in parallel 

been explained. Thus, the first element of the first vector and 60 on multiple resolutions. This training is described in the 

the element in the first row and first column of the matrix can flow-chart of FIG. 4a, 

be thought of as a pair. Correspondingly, the first element of The second set of images is used to train the fuzzy 

the second vector and the element in the second row and first combination procedure. This set of faces requires new or 

column of the vector can be thought of as a pair and so on different images of the objects or faces in the first set as well 

for the other three vectors and rows in the diagram. 65 as images of different objects or faces not present in the first 

The first operation to be performed is the sorting of these training set. The recognition rate and false access rates of the 

pairs into increasing order according to the magnitude of the classification systems comprising the local autocorrelation 
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feature vector extraction procedures, the linear discriminant 
analysis procedures and the recognition procedures are 
calculated and combined using the formula (22). The rec- 
ognition procedure is that in which the minimum of the 
distances is used for recognition and thresholding of the 5 
distances is used for rejection or alternatively those variants 
of this which have already been explained. This is equivalent 
to running multiple copies of the recognition system 
described in the prior art at multiple resolutions to gain a 
measure of the ability of these systems to recognize par- 10 
ticular faces at particular resolutions. This step is as shown 
in the flow-chart of FIG. 4b. 

Finally, a third set of faces for testing purposes is required. 
There may be only one face in this third set, although 
multiple copies of this image at the different resolutions are 15 
required. However, for the purposes of collecting data about 
the performance of the system, it is helpful if this third set 
contains a third set of images of the objects in the first set as 
well as other images of objects which have not been previ- 
ously seen by any of the systems, all of these images being 20 
present at the required multiple resolutions. The multiple 
sets of local autocorrelation feature extraction procedures 
followed by linear discriminant analysis procedures are run 
in parallel on the third set of images generating vectors of 
distances between the cluster centers of the known images 25 
and the unknown images. The following procedure is per- 
formed for each distinct image in the third set. Features 
generated from images of the same objects at the same 
orientation but different resolutions are combined by using 
fuzzy integration of the Choquet or Sugeno type, the fuzzy 30 
measure being employed being the Sugeno measure calcu- 
lated from the training data obtained from the second 
training set. This combination involves a modification of the 
distances into classifying numbers as previously described 
and graphs of performance of the combined system are 35 
useful for discovering the correct user supplied scaling 
parameter to be used. The combined data is then fed into a 
recognition procedure of the same type as used in the second 
stage, the combined fuzzy integrals possibly being modified 
in the manner previously described to permit operation with 40 
the recognition procedure which was also previously 
describes. This stage of the procedure in which the third set 
of images are recognized is not given in the prior art and 
requires a method of training, a method of combination and 
a means of converting the classification distances into a form 45 
in which they can be used for combination. This final 
recognition step is as shown, in the flowchart of FIG. 4c. 

The results of recognition can be displayed on a screen or 
a display or output to an appropriate control apparatus which 
may take one of the following forms but is not limited 50 
thereto. The results of the recognition procedure may be 
directed to controlling apparatus which: 

a) is used for recognition of faces to permit the entry of 
the persons belonging to those faces to some secure 
area; 55 

b) is used for the recognition of faces to permit the 
gathering of information concerning the whereabouts 
of persons attached to those faces by tracking their 
movements; 

c) is used to compare a face captured and digitized by 60 
some image capturing means with an additional sup- 
plied image of that face such as might be inscribed or 
printed on a credit card or other document or item, the 
ownership of which need to be checked by machine. 

In the case where the objects to be recognized are not 65 
faces, the controlling apparatus may have other functions. 
The controlling apparatus may be used for: 



a) the recognition of those objects to permit sorting; 

b) the recognition of those objects and their spatial 
alignment to permit manipulation in order to change 
their alignment; 

c) the grading of objects of the same type with respect to 
some visual quality such as size. 

These represent only a selection of possible controlling 
apparatus and these lists are not meant to restrict the 
incorporation of the process into other controlling apparatus 
as may be appropriate. 

As described above, according to the present invention, a 
method of recognizing faces comprises several steps applied 
to three sets of images of faces. In the first step, the first set 
a images is used to train the invention to recognize the 
known faces of the first set at multiple resolutions, mapping 
matrices being generated by linear discriminant analysis 
from the local autocorrelation features representing the 
images at each of the resolutions. In other words, one 
mapping matrix will be generated for each resolution from 
all of the features calculated at that resolution. In the second 
step, the mapping matrices are applied to the local autocor- 
relation features for each of the images in the second set at 
the resolutions at which the mapping matrices were 
generated, at least some of these images being of persons 
also present in the first step and at least one image of each 
of the known persons being present in the second set. At 
each of the resolutions which are to be combined, the 
distances between the mapped averages of the features for 
each of the known persons at that resolution are calculated 
and a particular measure of their performance is used to 
calculate fuzzy densities and hence a fuzzy measure which 
can be used in the next step for combining the results. In the 
third step, the third data set is used for testing. Each image 
is treated as an unknown image and is classified without 
additional knowledge by the invention. A multiplicity of 
images can be used to measure the statistical performance of 
the system for comparison with other systems but only a 
single image (at multiple resolutions) need be used. The 
image, or images are classified at multiple resolutions by 
using the technique of local autocorrelation coefficients as 
before. The previously determined mapping matrices appro- 
priate to the resolutions at which the characterizing features 
have been produced are applied to transform the character- 
izing feature vectors for the unknown image or images. 
Subsequently, the distance between the feature vectors for 
the unknown image or images and the cluster centers, which 
are the mapped averaged features corresponding to known 
persons, are calculated for each resolution. These distances 
are then converted into scores or pseudo-probabilities to be 
taken as the measurements produced by the sources which 
are the mapping matrices at each resolution. The distances, 
which have been converted into scores or pseudo- 
probabilities, are then combined using the Sugeno or 
Choquet integrals calculated with respect to a Sugeno or 
other measure. If the Sugeno measure is used, the fuzzy 
densities used to calculate the fuzzy measure are derived 
from the recognition rates and an error measure combined in 
a particular way as previously described. The result of this 
will be a series of scores, one for each known face and each 
classifier which describe the similarity or differences 
between the known faces and each of the unknown faces in 
the third test set. Recognition can then be performed by 
choosing the largest score, or smallest distance or some 
other means of choosing which of the known faces is most 
similar to each of the unknown faces with thresholding 
being employed to decide which faces are to be considered 
unknown. 
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Accordingly, the reader will see that this face or object 
recognition system can be used to recognize images of faces 
(or other objects) from a limited number of training images, 
these images not being required to be precisely aligned, 
scaled or normalized. The invention does not require that the 5 
faces (or other objects to be recognized) are localized within 
the image, does not require specialized images of faces, or 
of parts of faces or body parts for recognition, images of 
heads and upper torsos taken in a variety of poses and under 
no special lighting conditions sufficing to perform recogni- 1Q 
tion. Indeed, having a variety of training images serves to 
improve the reliability of the invention. 

Furthermore, the invention has the additional advantages 
that: 

a) it can perform recognition without the requirement of 
using color images; 15 

b) it can perform recognition using multiple resolutions in 
which the images at lower resolution can be derived 
from those at higher resolution by pixel averaging; 

c) it can be implemented, in part at least, by using a neural 
network or a plurality of neural networks; and 

d) it does not require time-ordered sequences of images 
during its training or test stages although multiple 
images are required. 

Although the description above contains many specifities, 
these should not be construed as limiting the scope of the 25 
invention but as merely providing illustrations of some of 
the presently preferred embodiments of this invention. For 
example, other methods of calculating the fuzzy densities 
which denote the importances of each of the resolutions can 
be employed, color images may be characterized, the 30 
method being applied separately to each of the color com- 
ponents and so on. 

Thus the scope of the invention should be determined by 
the appended claims and their legal equivalents, rather than 
by the examples given. 35 

What is claimed is: 

1. A process for recognizing known images of a plurality 
of objects, the process comprising: 

(a) obtaining a plurality of different resolutions for an 
unknown image; 40 

(b) performing local autocorrelation for each resolution of 
the unknown image to produce a feature vector for each 
resolution; 

(c) performing linear discriminant analysis for each fea- 
ture vector, wherein each feature vector from the local 45 
autocorrelation is processed employing a transforma- 
tion matrix, the transformation matrix being calculated 
with respect to known images required to be 
recognized, the linear discriminant analysis producing 

a plurality of outputs; 50 

(d) performing fuzzy integration to merge the outputs 
from the linear discriminant analysis to produce a 
merged output for each of the known images, each 
merged output being indicative of similarity of the 
unknown image to each known image; 55 

(e) analyzing each merged output to identify a known 
image corresponding to the unknown image. 

2. The process according to claim 1 including calculating 
the transformation matrix using a neural network. 

3. The process according to claim 2 wherein the known 60 
images are images of faces. 

4. The process according to claim 1 wherein performing 
fuzzy integration comprises performing Sugeno fuzzy inte- 
gration. 

5. The process according to claim 4 wherein performing 65 
fuzzy integration comprises calculating a Sugeno fuzzy 



6. The process according to claim 4 wherein the known 
images are images of faces. 

7. The process according to claim 5 wherein the known 
images are images of faces. 

8. The process according to claim 1 wherein obtaining a 
plurality of different resolutions of an unknown image 
comprises obtaining multiple copies of the unknown image 
using an image gathering means. 

9. The process according to claim 1 wherein obtaining a 
plurality of different resolutions of an unknown image 
comprises applying pixel averaging to the unknown image. 

10. A process for evaluating the similarity of a plurality of 
collections of images and for classifying the collections of 
images with respect to a known collection of images of 
objects of the same class, each of the collections containing 
only images of the same object produced at different orien- 
tations and resolutions, the resolutions being known for each 
image and being the same between the collections of images 
thereby forming sets of images at differing resolutions, the 
process comprising: 

(a) obtaining a plurality of different resolutions of an 
image to produce a collection of images to be classi- 
fied; 

(b) providing a plurality of first modules implementing 
local autocorrelation coefficient feature extraction and 
providing a plurality of second modules implementing 
linear discriminant analysis and equal in number to the 
first modules, each of the first modules providing an 
output to each of the second modules to form pairs of 
modules, the second modules storing information char- 
acterizing a collection of known images of objects of 
the same class, the second modules providing informa- 
tion on similarity of images from the collection of 
images to be classified to the collection of known 
images, the pairs of local autocorrelation feature analy- 
sis modules and linear discriminant analysis modules 
operating only at resolutions of known and predeter- 
mined accuracy for both images of the collection of 
images to be classified and the collection of known 
images; 

(c) providing a third module for integrating outputs from 
the pairs of local autocorrelation feature extraction 
modules and linear discriminant analysis modules, and 
for merging information from the pairs of modules 
operating at multiple resolutions, the third module 
performing fuzzy integration; 

(d) training the third module, including measuring the 
performance of the pairs of local autocorrelation fea- 
ture extraction modules and linear discriminant analy- 
sis modules to provide performance measurement; 

(e) combining the output of the pairs of local autocorre- 
lation feature extraction modules and linear discrimi- 
nant analysis modules based on the performance mea- 
surement; 

(f) producing an output similar to an output produced by 
one of the pairs of local autocorrelation coefficient 
feature extraction modules and linear discriminant 
analysis modules; and 

(g) providing a fourth module for analyzing the output 
from the third module such that recognition of images 
from the collection of images to be classified or rejec- 
tion of the collection of images to be classified is 
effected relative to the images from the collection of 
known images, an output of the fourth module being an 
identification of each of the images from the collection 
of images to be classified as being similar to images 
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from the collection of known images or as being 
different from the images in said collection of known 
images, wherein the pairs of local autocorrelation fea- 
ture extraction modules and linear discriminant analy- 
sis modules in conjunction with the third module and 
the fourth module permit recognition or rejection of the 
images from the collection of images to be classified, 
the recognition or rejection being with respect to the 
collection of known images, the recognition or rejec- 
tion having higher performance measurement than a 
performance measurement of any one of the pairs of 
local autocorrelation feature extraction modules and 
linear discriminant analysis modules in conjunction 
with one of the modules for implementing rejection or 
recognition. 

11. The process according to claim 10 wherein the pairs 
of local autocorrelation feature extraction modules and 
linear discriminant analysis modules are implemented by at 
least one neural network. 

12. The process according to claim 10 wherein the col- 
lection of known images are images of faces. 

13. The process according to claim 11 wherein the col- 
lection of known images are images of faces. 

14. The process according to claim 10 wherein the third 
module is implemented by Sugeno integration. 

15. The process according to claim 11 wherein the third 
module is implemented by Sugeno integration. 

16. The process according to claim 14 wherein a fuzzy 
measure employed in the fuzzy integration is a Sugeno fuzzy 
measure. 

17. The process according to claim 15 wherein a fuzzy 
measure employed in the fuzzy integration is a Sugeno fuzzy 
measure. 

18. The process according to claim 16 wherein the col- 
lection of images to be classified are images of faces. 
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19. The process according to claim 17 wherein the col- 
lection of images to be classified are images of faces. 

20. A process for recognizing an unknown image with 
respect to sets of known stored images comprising: 

5 (a) obtaining a plurality of different resolutions of the 
unknown image; 

(b) performing location autocorrelation coefficient feature 
extraction followed by linear discriminant analysis for 

10 each resolution of the unknown image to produce a 
plurality of outputs; 

(c) combining the plurality of outputs into groups for each 
set of known stored images using fuzzy integration and 
producing a classifying number for each group; 

15 (d) performing recognition or acceptance of the unknown 
image by examining the classifying number and com- 
paring the classifying number to a threshold; and 
(e) training the step of combining such that the plurality 

20 of outputs for the plurality of resolutions are combined 
in a manner other than averaging. 

21. The process according to claim 20 wherein perform- 
ing local autocorrelation includes using at least one neural 
network. 

25 22. The process according to claim 21 wherein combining 
the outputs includes performing Sugeno fuzzy integration 
using a Sugeno fuzzy measure. 

23. The process according to claim 21 wherein the 
unknown image is an image of a face. 
30 24. The process according to claim 22 wherein the 
unknown image is an image of a face. 

25. The process according to claim 22 wherein the 
unknown image comprises an image of a machine part. 
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